Conventionally, there are game systems using television receivers (hereinafter, simply abbreviated as “televisions”) as display devices. An example of such conventional game systems is a game system including a television and a game apparatus connected to the television. In this game system, game images generated by the game apparatus are displayed on the television and game sounds generated by the game apparatus are outputted from speakers of the television.
In the case of the game systems using televisions as display devices, there might be delays in displaying game images. Specifically, for the purpose of, for example, enhancement of the quality of images, recent digital televisions subject input images to various types of video processing, and display video-processed images. The video processing is generally time-consuming, and therefore, there might be a delay between the game apparatus outputting a game image to the television and the television displaying the game image. Accordingly, game systems using televisions as display devices have a problem of poor response of game displays to game operations.
Therefore, the present specification discloses a delay measurement system and method for measuring a delay in outputting an image or sound to a display device such as a television. The present specification also discloses a game apparatus, an information processing apparatus, a storage medium having a game program or information processing program stored therein, a game system, and an image display method which solve or reduce any problem to be caused by the delay as mentioned above.
(1) An example game system described herein includes a game apparatus and a portable display device.
The game apparatus includes an image generation section, an image output section, an image compression section, and an image transmission section. The image generation section sequentially generates first game images and second game images on the basis of a game process. The image output section sequentially outputs the first game images to a predetermined display device different from the portable display device. The image compression section sequentially compresses the second game images to sequentially generate compressed image data. The image transmission section sequentially transmits the compressed image data to the portable display device in a wireless manner.
The portable display device includes an image pickup section, an image reception section, an image decompression section, a display section, and an imaging data transmission section. The image reception section sequentially receives the compressed image data from the game apparatus. The image decompression section sequentially decompresses the compressed image data to acquire the second game images. The display section sequentially displays the second game images acquired through the decompression by the image decompression section. The imaging data transmission section transmits compressed imaging data to the game apparatus, the compressed imaging data being obtained by compressing a pickup image acquired by the image pickup section.
The game apparatus further includes an imaging data reception section, an image determination section, an image delay calculation section, and an image transmission control section. The imaging data reception section receives and decompresses the compressed imaging data to acquire the pickup image. The image determination section determines whether or not the pickup image acquired through the decompression by the imaging data reception section includes a predetermined first game image. When the pickup image is determined to include the predetermined first game image, the image delay calculation section calculates an image delay time between the image output section outputting the predetermined first game image and the predetermined display device displaying the predetermined first game image, on the basis of the time of the determination, the time of the output of the predetermined first game image by the image output section, and a processing time between the acquisition of the pickup image by the image pickup section and the determination. The image transmission control section delays the timing of transmitting the second game images to the portable display device, on the basis of the image delay time.
The “game apparatus” may be any information processing apparatus which performs a game process and generates a game image on the basis of the game process. The game apparatus may be an information processing apparatus for game use or a multipurpose information processing apparatus such as a general personal computer.
The “portable display device” corresponds to a terminal device 7 in an example embodiment to be described later, and also encompasses any device provided with features as mentioned above. Note that the term “portable” is intended to mean a size that allows the player to hold and move the device or arbitrarily change the position of the device.
The “game system” includes a game apparatus and a portable display device, and may or may not include a predetermined display device for displaying first game images. That is, the game system may or may not be provided in the form which includes the predetermined display device.
The “predetermined display device” encompasses any display device, such as a television in the example embodiment to be described later, which displays an input image after subjecting the image to some video processing.
The “predetermined first game image” is one of the first game images that is used for calculating the image delay time. Note that the concrete content of the predetermined first game image may be arbitrary, and the predetermined first game image may be a still image or one of a set of dynamic images.
According to the above configuration (1), the portable display device picks up a first game image displayed on the predetermined display device, and the pickup image is used to calculate the image delay time. As a result, a delay is measured on the basis of an actually displayed image, so that the image delay time can be accurately calculated. In addition, the game apparatus delays the timing of transmitting the second game images to the portable display device on the basis of the image delay time, so that the timing of displaying the game images can be equalized between the predetermined display device and the portable display device. Thus, according to the above configuration (1), it is possible to synchronize the timing of displaying game images between the predetermined display device and the portable display device, making it possible to solve any problem due to a display delay of the predetermined display device.
(2) The predetermined display device may include a speaker.
In this case, the game apparatus further includes a sound generation section, a sound output section, and a sound transmission section. The sound generation section generates a first game sound and a second game sound on the basis of the game process. The sound output section outputs the first game sound to the predetermined display device. The sound transmission section wirelessly transmits the second game sound to the portable display device.
The portable display device includes a sound reception section, a speaker, a microphone, and a detected sound transmission section. The sound reception section receives the second game sound from the game apparatus. The speaker outputs the second game sound. The detected sound transmission section transmits a sound detected by the microphone to the game apparatus.
The game apparatus further includes a detected sound reception section, a sound determination section, a sound delay calculation section, and a sound transmission control section. The detected sound reception section receives the detected sound. The sound determination section determines whether or not the detected sound received by the detected sound reception section includes a predetermined first game sound. When the detected sound is determined to include the predetermined first game sound, the sound delay calculation section calculates a sound delay time between the sound output section outputting the predetermined first game sound and the speaker of the predetermined display device outputting the predetermined first game sound, on the basis of the time of the determination, the time of the output of the predetermined first game sound by the sound output section, and a processing time between the detection of the sound by the microphone and the determination. The sound transmission control section delays the timing of transmitting the second game sound to the portable display device, on the basis of the sound delay time.
According to the above configuration (2), the portable display device detects a first game sound outputted to the predetermined display device, and the detected sound is used to calculate the sound delay time. As a result, a delay is measured on the basis of an actually outputted sound, so that the image delay time can be accurately calculated. In addition, the game apparatus delays the timing of transmitting the second game sound to the portable display device on the basis of the sound delay time, so that the timing of outputting the game sounds can be equalized between the predetermined display device and the portable display device. Thus, according to the above configuration (2), it is possible to synchronize the timing of outputting game sounds between the predetermined display device and the portable display device, making it possible to solve any problem due to a sound output delay of the predetermined display device.
(3) The image generation section may generate as the first game image or the second game image an image making a notification for prompting to direct the image pickup section of the portable display device toward a screen of the predetermined display device, and after the image is displayed on the predetermined display device or the portable display device, the image generation section may generate and display the predetermined first game image on the predetermined display device.
According to the above configuration (3), a notification is made for prompting to direct the image pickup section of the portable display device toward the screen of the predetermined display device before the predetermined first game image for use in measuring the image delay time is displayed. As a result, it is possible to prevent the image pickup section from not being directed toward the screen of the predetermined display device when the predetermined first game image is displayed, making it possible to prevent the image delay time from being measured inaccurately.
(4) The game apparatus may further include a storage section capable of storing a plurality of second game images. In this case, the image transmission control section stores the second game images generated by the image generation section to the storage section, and causes the image transmission section to transmit the second game images after a lapse of a standby time based on the image delay time since their generation.
The “storage section” may be any storage, such as a main memory or flash memory in the example embodiment to be described later, which can be accessed by the game apparatus. Note that the second game images to be stored in the storage section may be image data compressed by the image compression section or image data to be compressed later.
According to the above configuration (4), the game apparatus has the second game images stored therewithin, and transmits the second game images after a lapse of a standby time. Thus, it is possible to readily adjust the timing of transmitting the second game images.
(5) The image transmission control section may calculate a standby time until the second game image is transmitted to the portable display device, on the basis of the image delay time, and a time period between the second game image being generated and the portable display device displaying the second game image.
According the above configuration (5), the standby time is determined considering not only the image delay time but also the time period between the second game image being generated and the portable display device displaying the second game image. Thus, it is possible to accurately synchronize the timing of displaying the first game image with the timing of displaying the second game image.
(6) Another example game system described herein includes a game apparatus and a portable display device.
The game apparatus includes a sound generation section, a sound output section, and a sound transmission section. The sound generation section generates a first game sound and a second game sound on the basis of a game process. The sound output section outputs the first game sound to a predetermined display device different from the portable display device, the predetermined display device including a speaker. The sound transmission section wirelessly transmits the second game sound to the portable display device.
The portable display device includes a microphone, a sound reception section, a speaker, and a detected sound transmission section. The sound reception section receives the second game sound from the game apparatus. The speaker outputs the second game sound. The detected sound transmission section transmits a sound detected by the microphone to the game apparatus.
The game apparatus further includes a detected sound reception section, a sound determination section, a delay calculation section, and a sound transmission control section. The detected sound reception section receives the detected sound. The sound determination section determines whether or not the detected sound received by the detected sound reception section includes a predetermined first game sound. When the detected sound is determined to include the predetermined first game sound, the delay calculation section calculates a sound delay time between the sound output section outputting the predetermined first game sound and the speaker of the predetermined display device outputting the predetermined first game sound, on the basis of the time of the determination, the time of the output of the predetermined first game sound by the sound output section, and a processing time between the detection of the sound by the microphone and the determination. The sound transmission control section delays the timing of transmitting the second game sound to the portable display device, on the basis of the sound delay time.
The “predetermined first game sound” is one of the first game sounds that is used for calculating the sound delay time. Note that the concrete content of the predetermined first game sound may be arbitrary.
According to the above configuration (6), as in the above configuration (2), the portable display device detects a first game sound outputted to the predetermined display device, and the detected sound is used to calculate the sound delay time. As a result, a delay is measured on the basis of an actually outputted sound, so that the image delay time can be accurately calculated. In addition, the game apparatus delays the timing of transmitting the second game sound to the portable display device on the basis of the sound delay time, so that the timing of outputting the game sounds can be equalized between the predetermined display device and the portable display device. Thus, according to the above configuration (6), it is possible to synchronize the timing of outputting game sounds between the predetermined display device and the portable display device, making it possible to solve any problem due to a sound output delay of the predetermined display device.
(7) An example delay measurement system described herein measures a display delay of a predetermined display device for displaying an input image after subjecting the image to predetermined video processing. The delay measurement system includes an information processing apparatus capable of communicating with the predetermined display device and a portable terminal device capable of wirelessly communicating with the information processing apparatus.
The information processing apparatus includes an image generation section and a first image output section. The image generation section generates a predetermined image. The first image output section outputs the predetermined image to the predetermined display device.
The terminal device includes an image pickup section and a pickup image transmission section. The pickup image transmission section transmits a pickup image acquired by the image pickup section.
The information processing apparatus further includes a pickup image reception section, an image determination section, and an image delay calculation section. The pickup image reception section receives the pickup image. The image determination section determines whether or not the pickup image includes the predetermined image. When the pickup image is determined to include the predetermined image, the image delay calculation section calculates an image delay time between the first image output section outputting the predetermined image and the predetermined display device displaying the predetermined image, on the basis of the time of the determination, the time of the output of the predetermined image by the first image output section, and a processing time between the acquisition of the pickup image by the image pickup section and the determination.
The “information processing apparatus” may be any information processing apparatus provided with the function of generating images. For example, the “information processing apparatus” may be a game apparatus as in the example embodiment to be described later or may be a multipurpose information processing apparatus such as a general personal computer.
The “portable terminal device” is a terminal device provided with the function of displaying images and functioning as an operating device in the example embodiment to be described later, but the “portable terminal device” may be any device provided with features as mentioned above. Note that the term “portable” is intended to mean a size that allows the player to hold and move the device or arbitrarily change the position of the device.
The “delay measurement system” includes an information apparatus and a terminal device, and may or may not include a predetermined display device for displaying first images. That is, the delay measurement system may or may not be provided in the form which includes the predetermined display device.
The “predetermined image” is an image for use in calculating the image delay time. Note that the concrete content of the predetermined image may be arbitrary, and the predetermined image may be a still image or one of a set of dynamic images.
According to the above configuration (7), the portable display device picks up a predetermined image displayed on the predetermined display device, and the pickup image is used to calculate the image delay time. As a result, a delay is measured on the basis of an actually displayed predetermined image, so that the image delay time can be accurately calculated.
(8) The image generation section may sequentially generate first images to be displayed on the predetermined display device and second images to be displayed on the terminal device. In this case, the first image output section sequentially outputs the first images to the predetermined display device. The information processing apparatus further includes a second image output section for sequentially outputting the second images to the terminal device.
In addition, the terminal device further includes an image reception section and a display section. The image reception section sequentially receives the second images from the information processing apparatus. The display section sequentially displays the received second images.
The information processing apparatus further includes an image output control section for delaying the timing of outputting the second images to the terminal device, on the basis of the image delay time.
According to the above configuration (8), the information processing apparatus delays the timing of transmitting the second images to the portable display device, on the basis of the image delay time, so that the timing of displaying the images can be equalized between the predetermined display device and the portable display device. Thus, according to the above configuration (8), it is possible to synchronize the timing of displaying images between the predetermined display device and the portable display device, making it possible to solve any problem due to a display delay of the predetermined display device.
(9) The information processing apparatus may further include a sound generation section, a first sound output section, and a second sound output section. The sound generation section generates a first sound to be outputted to the predetermined display device and a second sound to be outputted to the terminal device. The first sound output section outputs the first sound to the predetermined display device. The second sound output section outputs the second sound to the terminal device.
In addition, the terminal device further includes a sound reception section and a speaker. The sound reception section receives the second sound from the information processing apparatus. The speaker outputs the second sound.
The information processing apparatus further includes a sound output control section for delaying the timing of outputting the second sound to the terminal device, on the basis of the image delay time.
According to the above configuration (9), the timing of outputting the second sound to the terminal device is adjusted on the basis of the image delay time. Thus, it is possible to synchronize both the timing of displaying images and the timing of outputting sounds between the predetermined display device and the portable display device. Moreover, according to the above configuration (9), it is not necessary to measure the sound delay time, and therefore, it is possible to achieve synchronization in terms of both images and sounds with a simplified process.
(10) The predetermined display device may include a speaker. In this case, the information processing apparatus further includes a sound generation section and a sound output section. The sound generation section generates a predetermined sound. The sound output section outputs the predetermined sound to the predetermined display device.
The terminal device includes a microphone and a detected sound transmission section for transmitting a sound detected by the microphone to the information processing apparatus.
The information processing apparatus further includes a detected sound reception section, a sound determination section, a sound delay calculation section, and an output control section. The detected sound reception section receives the detected sound. The sound determination section determines whether or not the detected sound received by the detected sound reception section includes the predetermined sound. When the detected sound is determined to include the predetermined sound, the sound delay calculation section calculates a sound delay time between the sound output section outputting the predetermined sound and the speaker of the predetermined display device outputting the predetermined sound, on the basis of the time of the determination, the time of the output of the predetermined sound by the sound output section, and a processing time between the detection of the sound by the microphone and the determination. The output control section delays at least one of the timing of outputting a sound by the sound output section and the timing of outputting an image by the image output section on the basis of the image delay time and the sound delay time, such that synchronization is achieved between display of the image on the predetermined display device and output of the sound from the speaker of the predetermined display device.
According to the above configuration (10), the information processing apparatus calculates an image delay time and a sound delay time, and on the basis of these delay times, the information processing apparatus adjusts the timing of displaying images on the predetermined display device and the timing of outputting sounds from the speaker of the predetermined display device. Thus, it is possible to synchronize these two timings in the predetermined display device, making it possible to prevent the user from being provided with a feeling of unnaturalness due to a display delay or sound output delay of the predetermined display device.
(11) Another example delay measurement system described herein measures a sound output delay of a speaker included in a predetermined display device. The delay measurement system includes an information processing apparatus capable of communicating with the predetermined display device and a portable terminal device capable of wirelessly communicating with the information processing apparatus.
The information processing apparatus includes a sound generation section and a first sound output section. The sound generation section generates a predetermined sound. The first sound output section outputs the predetermined sound to the predetermined display device.
The terminal device includes a microphone and a detected sound transmission section for transmitting a sound detected by the microphone to the information processing apparatus.
The information processing apparatus further includes a detected sound reception section, a sound determination section, and a sound delay calculation section. The detected sound reception section receives the detected sound. The sound determination section determines whether or not the detected sound received by the detected sound reception section includes the predetermined sound. When the detected sound is determined to include the predetermined sound, the sound delay calculation section calculates a sound delay time between the sound output section outputting the predetermined sound and the speaker of the predetermined display device outputting the predetermined sound, on the basis of the time of the determination, the time of the output of the predetermined sound by the sound output section, and a processing time between the detection of the sound by the microphone and the determination.
According to the above configuration (11), the portable display device detects a predetermined sound outputted from the speaker of the predetermined display device, and the detected sound is used to calculate the sound delay time. As a result, a delay is measured on the basis of an actually outputted predetermined sound, so that the sound delay time can be accurately calculated.
(12) The sound generation section may generate a first sound to be outputted to the predetermined display device and a second sound to be outputted to the terminal device. In this case, the first sound output section outputs the first sound to the predetermined display device. The information processing apparatus further includes a second sound output section for outputting the second sound to the terminal device.
The terminal device further includes a sound reception section and a speaker. The sound reception section receives the second sound from the information processing apparatus. The speaker outputs the received second sound.
The information processing apparatus further includes a sound output control section for delaying the timing of outputting the second sound to the terminal device, on the basis of the sound delay time.
According to the above configuration (12), the information processing apparatus delays the timing of transmitting the second sound to the portable display device, on the basis of the sound delay time, so that the timing of outputting sounds can be equalized between the predetermined display device and the portable display device. Thus, according to the above configuration (12), it is possible to synchronize the timing of outputting sounds between the predetermined display device and the portable display device, making it possible to solve any problem due to a sound output delay of the predetermined display device.
(13) The information processing apparatus may further include an image generation section, a first image output section, and a second image output section. The image generation section sequentially generates first images to be displayed on the predetermined display device and second images to be displayed on the terminal device. The first image output section sequentially outputs the first images to the predetermined display device. The second image output section sequentially outputs the second images to the terminal device.
The terminal device further includes an image reception section and a display section. The image reception section sequentially receives the second images from the information processing apparatus. The display section sequentially displays the received second images.
The information processing apparatus further includes an image output control section for delaying the timing of outputting the second images to the terminal device, on the basis of the sound delay time.
According to the above configuration (13), the timing of outputting the second images to the terminal device is adjusted on the basis of the sound delay time. Thus, it is possible to synchronize both the timing of outputting sounds and the timing of displaying images between the predetermined display device and the portable display device. Moreover, according to the above configuration (13), it is not necessary to measure the image delay time, and therefore, it is possible to achieve synchronization in terms of both images and sounds with a simplified process.
The present specification discloses example information processing apparatuses (game apparatuses) provided with features (excluding the image pickup section) of the game systems or delay measurement systems described in (1) to (13) above. Moreover, the present specification also discloses example non-transitory computer-readable storage media each having stored therein a game program or information processing program for causing computers of the information processing apparatuses to function as means equivalent to the features mentioned above. Furthermore, the present specification also discloses examples of an image display method, a sound output method, and a delay measurement method as performed in the game systems or delay measurement systems described in (1) to (13) above.
The game systems, delay measurement systems, etc., as described above, make it possible to calculate a display delay and/or a sound output delay in a predetermined display device. Moreover, by using the calculated delay(s), it is rendered possible to achieve synchronization between the predetermined display device and a portable display device and synchronization between image display and sound output in the predetermined display device.
These and other objects, features, aspects and advantages will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.